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ABSTRACT 



This paper explores the use of student evaluation of 



teaching (SET) as part of the quality assurance cycle and suggests an 
alternative approach to evaluate the provision of teaching in tertiary 
institutions. SETs usually involve questionnaires that ask students 
anonymously to rate the quality of teaching on a 4- or 5 -point scale. SETs 
have been used in universities for more than 30 years to assess the quality 
of teaching, but the use of SET instruments had often been accompanied by 
counter-productive effects. Using them for proportion and tenure decisions 
had contributed to the lowering of academic standards. An alignment method of 
teacher evaluation yields better results. The alignment method considers 
skills, understanding, and attitudes, and whether the current state of these 
change attributes is closely aligned with the ideal state. Aligning changes 
expected by the lecturer with changes expected by the students results in 
improved teaching, and the evaluation of the alignment serves as an indicator 
of teacher effectiveness. (SLD) 
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Improving the Quality of Tertiary Education Through Student Evaluation of Teaching 

Tony Bastick 

University of the West Indies 
tbastick@uwimona.edu.jm 

This paper evaluates the use of Student Evaluations of Teaching (SETs) as part of the Quality Assurance 
Cycle, finds this instrument to be inappropriate and suggests an alternative instrument to evaluate the 
provision of teaching in tertiary institutions. 

Figure 1 : The Quality Assurance Cycle using Student Evaluation of Teaching (SETs) 
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Figure 1 illustrates the Quality Assurance Cycle for evaluating institutional provisions. Where that 
provision is Teaching, it is common to use SETs as the main evaluative instrument. These instruments 
are usually questionnaires that ask students to anonymously rate the quality of teaching on a 4 or 5 point 
Likert scale from strongly disagree to strongly agree. Apart from the on-going Quality Assurance Cycle 
it is occasionally necessary to take a Quality Audit of the Quality Assurance procedures themselves in 
order to ensure that these procedures continue to be effective. This paper contributes to the Quality 
Audit of the SET procedure for assessing teaching provision. 

SETs have been used in universities for more than thirty years to assess the quality of teaching and as an 
indicator of successful teaching for promotion and tenure decisions. Unfortunately, their use has been 
accompanied by many counter- productive effects such as discouraging innovation, and deterring 
instructors from challenging students (Damron, 1995; Murray, 1984; Ruskai, 1996). Although their 
outcomes are intended to improve teaching, a major negative effect of also using them for promotion 
and tenure decisions has been to contribute to the lowering of academic standards. In the copious 
literature on the subject, this effect is referred to as ‘grade inflation’ or ‘dumbing down courses’ and 
some universities who use SETs now make statistical adjustments for these effects (Gillmore, & 
Greenwald, 1999). SETs have become known as little more than ‘smile sheets’ measuring popularity 
and ‘customer satisfaction’ (Altschuler, 1999), and lecturers have developed many methods for 
improving their SET scores that do not necessary improve teaching (Crumbley, 1995). Its seems that one 
reason SETs continue to be used is that there has not been an expedient alternative. This paper 
introduces such an alternative - an alignment method. 
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Alignment method of assessment 

Many psychometric instruments use what I call ‘alignment methods’. In an alignment method a 
respondent’s current state is assessed and his/her ideal state is also assessed on the same indicators. The 
difference between the current and ideal states is the alignment. Where the difference is large, there is 
poor alignment which is indicative of problems. Where the difference is small, alignment is good which 
indicates that the current state is close to the ideal. Improved alignment can also be used as a measure of 
successful intervention strategies. What is crucial to the alignment method is the choice of indicators 
measured to assess the current and ideal states. In tertiary education there are three change attributes that 
are emphasized in quality teaching and learning. These are Skills, Understanding and Attitudes 
operationally defined as: 

1 . Skills - learning of facts/processes. Assessed by speedy accurate reproduction. 

2. Understanding - professional competence. Assessed by justification of novel application. 

3. Attitudes - professional values. Assessed by demonstration. 

The alignment method uses these three indicators. It should be noted that critical thinking is promoted 
by teaching and assessment of professional competence. This is because there are no right/wrong 
answers and only justifications are assessed. Alignment is not based on the course objectives. Course 
objectives and content are used as vehicles for emphasising the desired degrees of Skills, Understanding 
and Attitudes. This emphasis will vary according to the course level and culture of the subject taught. 

What are aligned are ‘changes expected by the lecturer’ and ‘changes expected by the students’ in each 
of these three attributes. Numerically stated: Alignment = changes expected by Lecturer - changes 
expected by students. Zero is the perfect score, the theory being that students achieve higher standards if 
they and their lecturer are working towards the same changes. The following Figure 2 shows the seven 
core questions that the lecturer and each student answers for the alignment to be calculated. 



Figure 2: Five minute assessment form 



Course Assessment - Skills, Competence and Attitudes 



(i) Emphasis on Skills 

j (getting it right) 



As it is now on this course 

lYour 

estimate 

lout of 100 



As it should be on this course 

lYour 

estimate 

lout of 100 



Write a 
Number j 
in each j 
box 



As it is now on this course 



(ii) Emphasis on Competence 

(biowing why) 



Your 
estimate 
out of 100 



i 



(iii) Emphasis on Attitudes 

(relevance to your life) 



As it is now on this course 

lYour 

estimate 

lout of 100 



As it should be on this course 

lYour 

estimate 

lout of 100 



Write a : 
Number j 
in each \ 
box \ 
l 



As it should be on this course 

lYour 

estimate 

lout of 100 



Write a 
Number 
in each 
box 



How much so far have 

you enjoyed your experience of 

the teaching on this course? 



Your Write a 
estimate Number 
out of 100 in the box 



These forms are confidential, not anonymous. In fact, students have to pass a test when they enroll so as 
to earn the right to be considered as informed assessors. Individual’s alignments can be grouped to 
assess the mean alignment of any group of interest - males v females, experienced v novice students, 
older v younger, option 1 v option 2 students, etc. 

Two alignment scores are calculated; 

• Alignment of Scope (changes in absolute quantity) and 

• Alignment of Proportions (changes in relative quantity) 

However, these formative alignment indicators, that are measured during the course, are only predictors 
of quality teaching. They are not the criteria of quality teaching. The two accepted summative criteria of 
quality teaching are: 

• Academic standards and 

• Enjoyment of learning 

Continued validation of the theory 

When the courses are over and the academic results are compared with the alignment scores, it is 
possible to validate the theory for each course, and for each group of students taking each course. 

• Alignment of Scope correlates with Academic standards and 

• Alignment of Proportions correlates with Enjoyment of learning 

Further, when the course has finished it is possible to use sensitivity analyses on the data to calculate 
those lecturer’s changes that would have most aligned the teaching and thus maximized the academic 
results and enjoyment of the students. It is seen from actual alignment data that having chosen these 
perferred changes would have increased the correlations between alignment and academic standards, 
thus futher validating the theory that alignments are predictors of quality teaching. 

Optimization of teaching 

Traditional SETs are a ‘post mortem’ assessment, collected at the end of the course when it is too late to 
feedback to help the students who made the assessments. However, a lecturer does not have to wait until 
the course is over to optimize teaching using the alignment method. The data collected in-course can be 
processed by the same type of sensitivity analysis to calculate the optimum changes that should be made 
by the lecturer to maximize the students post-course academic attainment and/or course enjoyment. 

Administrative Decision point assessment of quality teaching 

The lecturer may utilize the five-minute alignment form many times during the course to keep teaching 
on track. The administration uses it just once near the end of the course to calculate the final alignment 
score for that lecturer’s quality of teaching. This results in a single decision point number that can be 
compared across the institution and used for promotion and tenure decisions. 

Discussion 

This paper has only touched on the classroom assessment use of the Alignment Method. It has not 
discussed the staff and course development aspects of the method or the many benefits the method offers 
for Quality Assurance compared to traditional SETs. 




This alignment method of assessing teaching quality offers 10 main benefits: 

1 It identifies the quality of teaching experienced by each individual student. 

2 It can be used to identify groups of students that might be disadvantaged by the teaching. 

3 It offers detailed diagnostic reports to help the lecturer. 

4 It only takes 5 minutes to administer and the analysis is quick and low-cost. 

5 It can be given several times in-course resulting in optimum recommendations to keep teaching 
on track. 

6 It is sensitive to criteria considered important in different subject areas and by different Faculties 
and levels of students. 

7 It maintains lecturer/student trust and promotes higher quality teaching and higher quality 
learning. 

8 It protects academic freedom, is non-threatening and has built in protection for Faculty who 
teach intransigent students and difficult courses. 

9 It uses one standard form and gives one single decision-point number that can be used in 
institutional evaluations for comparing quality of teaching across the university e.g. for Quality 
Audits, teaching awards and for promotion and tenure decisions. 

10 Post-course correlations with academic standards evidence the reliability and validity of the 
instrument for each course and for subgroups of students taking each course on which it is used. 

This alignment method can be flexibly piloted at different levels within an institution - at the level of full 
institutional evaluation, at the level of staff and course development within Faculties, Schools or 
Departments and at the level of individual lecturers who are interested in improving the quality of their 
own teaching for their own students. Web-based Alignment software is now being developed that will 
enable lecturers and administrators from tertiary institutions world-wide to avail themselves of the 
benefits of using the alignment method in their own Quality Assurance Cycles. 
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